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Recovery of Ego-Motion Using Image Stabilization 
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Abstract 

A method for computing the 3D cajnera motion 
(the cgo-tnoiion) in a static scene is intToduced, which 
is based on computing the 2D image motion of a sin- 
gle image region directly from image intensities. The 
computed image motion of this image region is used to 
register the images SO thftt the detected image region 
appears stationary. The resulting displacement field 
for the entire scene between the registered £rame$ is af- 
fected only by the 3D translation of the camera. After 
canceliug the effects of the camera rotration by using 
such 2D image registration, the 3D camera translation 
is computed by finding the focus-of-expansion in the 
transl(^tion-only set of registered frames This step is 
followed by computing the camera rotation to coni- 
plete the computation of the ego-motion. 

The presented method avoids the inherent prob- 
lems in the computation of optical ^ovf and of feature 
matching I and does not assume any prior feature de- 
tection or feature correspondence. 

1 Introduction 

The motion observed in an image sequence can be 
caused by camera motion (ego-mot.ion) and by mo- 
lions of objects moving in the Scene. In this paper we 
address the case of a camera moving in a static scene. 
Complete 3D motion estimation is difficult since the 
image motion at every pixel depends, in addition to 
the six parameters of the camera motion, on the depth 
at the corresponding scene point. To overcome this 
difSculty, additional constraints are usually added to 
the motion model or to the environment model. 

3D motion is often estimated from the optical or 
normal flow derived between two frames [1, 12, 22], 
or &om the correspondence of distinguished features 

'This icsearch has been eponaoicdby the U.S Olfice ofNaval 
Rcseaxch under Gzani N0OOI4-93-1-12U2, R&T Pcoject Code 

f M. Ixazii is now with UAvfd Samoft ReBearch Centex. 



(points, lines, contours) extracted from successive 
frames [10, 13, 7] Both approaches depend on the 
accuracy of the feature detection, which can not al- 
ways be assured. Methods for computing the ego- 
motion direcily from image intensities were also sug- 
gested (11, 14]. 

Camera rotations and translations caJa induce sim- 
ilar image motions [2, 8] causing ambiguities in theu: 
interpretation. At depth discontinuities, however, it 
is much easier to dJetinguish between the effects of 
camera rotations and camera translations, as the im- 
age motion of neighboring pixels at diifcrent depths 
will have eimilar rotational components, but dilTer- 
ent tranalational components. Motion parallax meth- 
ods use this effect to obtain the 3D c^era motion. 
[18, 17, 7]. Otber methods use motion parallax for 
shape representation and analysis [23, 6, 9]. 

In this paper a method for computing the ego- 
motion directly from image intensities is introduced 
At ilrst only 2D image motion is extracted, and later 
this 2D motion is used to simplify the computation of 
the 3D ego-motion. 

We use previously developed methods [15, 16] to 
detect and track a single image region and to com- 
pute its 2D parametric image motion, ft is important 
to emphasize that the 3D camera motion cannot be 
recovered solely from the 2D parametric image mo- 
tion of a single image region » as there are a couple of 
such 3D interpretations [20]. Tt was shown that 3D 
motion of a plsuiar surface can be computed from its 
2D affine motion in the image s^d from the motion 
derivatives [21], but motion derivatives introduce sen- 
sitivity to noise- Moieover, the problem of recovering 
the 3D camera motion directly from the image motion 
field is an ill-conditioned problem, since small errors in 
the 2D flow field usually result in large perturbations 
in the 3D motion [2]. 

To overcome the difficulties and ambiguities in the 
computation of the ego-motion, introduce the fol- 
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lowing scheme: The first frame is warped towards the 
second frame using the computed 2D image motion &t 
the detected image region- This registration caacels 
the efFecta of the camera totation for the entire scene, 
aad the resulting image displacements between the 
two registered frames are due only to the 3D transla- 
tion of the camera. This translation is computed by lo- 
cating the FOE (fbcus-of-expanaion) between the two 
reji^ererf frames. Once the 30 translation ie known it 
can be used, together with the 2D motion parameiezs 
of the detected image region, to compute the 3D rota- 
tion of the cameia by solving a set of /in ear equations. 

The 2D image region registration technique used in 
this work allows easy decoupling of the translational 
and rotational motions, as only motion parallax infov- 
Diation remains after the registration. As opposed to 
othei methods using motion parallax [18, 19i 17, 7], 
our method does not rely on 2D motion information 
computed near depth discontinuities, where it is inac- 
curate, but on motion computed over an entire image. 
The effect of motion parallax is obtained at all Bcene 
points that are not located on the extension of the 3D 
surface which corresponds to the registered image re- 
gion. This gives dense parallax data, as these scene 
points need not be adjacent to the re^stered 3D sur- 
face 

The advantage of this technique is in its simplicity 
and in its robustness. No prior detection and matching 
ate assumed, it requires solving only small sets of lin- 
ear equations, and each comptiiational step is stated 
as an overdetermined problem which is numerically 
Stable. 

2 Ego-Motion from 2D Image Motion 

Jn this section we describe the technique for com- 
putinfi the 3D ego-mction given the 2D parametric 
motion of a single image region The method for au- 
tomatically computing the 2D motion of a single image 
region is briefly described in Sec. 4 

2.1 Basic Model and Notations 

Let (X,y,^) denote the Cartesian coordinates of 
a scene point with respect to the camera (see Fig. 1), 
and let (t, y) denote the corresponding coordinaies in 
the image plane- The image plane is located at the 
focal length: Z = /c- The perspective projection of 
a scene point P = (XtY, Zy on the image plane at a 
point p = (s>yy is expressed by. 



(1) 



The camera motion has two components: a translation 
'i'= (7i,7V,T^)* and a rotation = (Ox, fly, H^)*. 




Figure 1: The coordinate system. 
The coordinate system (J^ , V, Z) is attached to the cam- 
cta, and the cozrespojiding image cooxdinatca (z, y) on 
the image plajfte are located at ^ = A point P = 

[X^Yi Zy in the world is piojcclcd onto an image point 

lepiesent the rdative traaalaUon and lotation of the 
caueza in the scene. 

Due to the camera motion the scene point P = 
{X, y, Zy appears to be moving relative to the camera 
with rotation -0 and translation — T, and is therefore 
observed at new world coordinates P = (^ ,y" , Z 
expressed by: 





\ X' 1 




" X ' 




y' 




y 








z 



-T 



(2) 



where ilf-n is the matrix corresponding to a rotation 
by -fi. 

When the field of view is not very large and the 
camera motion has a relatively small rotation [1] , the 
2D displacement (it, v) of an image point («, y) in the 
image plane can be expressed by [20, 3]: 



(3) 



The following is noted from Eq, (3): 

• Since all translations are divided by the unknown 
depth only the direction of the translation can 
be recovered, but not its magnitude. 

• The contribution of the camera Toiaiioti to ^he 
displacement of an image point is indt.'ptndeni of 
the depth Z df the corresponding scene point. 

All points {X, y, Z) of a planar siirface in the 3D scene 
satisfy a plane equation Z = A'^B^X-^ y, which can 



455 



PA(£ 12/17 ' RCVD AT 1 1:20:18 AM [Eastern Dayflp r^^^ 



06/65/2006 10:20 FAX 6123051226 



MUETING RAASCH GEBHARDT 



Igl013 



be expressed in terms of image coordinates by using 
Eq. (1) 



(4) 



la a similar manipulatioB bo that in [1], substituting 
Bq. (4) in Eq- (3) yields: 




(5) 



b = 
d = 



-fcaTx-fctty 
ctTz - StffTx 



t 

f 

9 
h 



(6) 



in 



Eq. (5) describes the 2D parametric motion 
the image plane, expressed by eight parameters 
(o, 6, c, d, e, /, g, A), which corresponds to a general 3D 
motion of a planar surface in the scene, afisunning a 
small field of view and a small rotation- We call 
Eq. (5) a pseudo 2D projective transformation, since 
under these assumptions, it is a good approximation 
to the 2D projective transformation. 

2.2 General Framework of the AlgQtithm 

In this section we present a scheme which utilizes 
the robustness of the 2D motion computation for com- 
puting 3D motion between two consecutive frames: 

1. A single image region is automatically detected, 
and its 2D paramtitic image motion is computed 
(Sec. 4). 

2. The two frames arc registered according to the 
computed 2D parametric motion of the detected 
image region. This image region stabilization can- 
cels the rotational component of the camera mo- 
tion for the entire scene (Sec 2.3), and the cameia 
translation can now be computed from the focus- 
of-expansion between the two rcgisiertd ftamcs 
(Sec. 2.4), 

3. The 3D rotation of the camera js now computed 
(Sec. 2.6) given the 2D motion paiwneters of the 
detected image region and the 3D translation of 
the camera. 

2.3 Eliminating Camera Rotation 

At this stage we assume that a single image region 
with a parametric 2D image motion has been detected; 
and that the 2D image motion of that region has been 
computed. The automatic detection and computation 
of the 2D image motion for planar 3D surfaces is de- 
scribed in Sec. 4. 



Let (u(a:,T/), t?(«,i/)) denote the 2D image motion 
of the entire scene from &ame fi to frame /2, and 
let (usi^i y)} y)) <lenote the 2D image motion of 
a single image region (the deUcUd image region) be- 
tween the two frames. It was mentioned in Sec. 2 1 
that (ujjVtf) can be expressed by a 2D paxaraetric 
transformation in the image plane if the image re- 
gion is an image of a planar surface in the 3D scene 
(Eq. (5)). Let $ denote the 3D surface of the detected 
image region^ with depths Zt(x^y). Note that only 
the 2D motion pararoeteis {Ui{x^y)^Vg(x,y)) of the 
planar surface are known. The 3D poeition or motion 
parameters of the planar surface are still unknown. 
Let denote the frame obtained by warping the en- 
tire frame A towards frame /a according to the 2D 
parametric transformation (ti^, Vg) tzttnded to the en- 
tire frame. This warping will cause the image region 
of the detected planar surface, as well as scene parts 
which are coplanar with it, to be stationary between 
and /2. In the warping process, each pixel (ir,y) 
in fx is displaced by (u,(«, y), ^^(x, y)) to form fi". 
3D points that are not located on the surface s (i.e., 
2(x, y) ^ Zs{z, y)) will not be in registration between 
and /a. 

We will now show that the 2D image motion be- 
tween the registered frames, (ff and /a) is affected 
only by the camera translation J*. 

Let Pi = (Xj,yi,Zi)* denote the 3D scene point 
projected onto pi = (ri,yi)' in /i- According to 
Eq. (1): Pi = (a:ifj-j2/if^,^i)*- Due to the camera 
motion (0,r) from liame f\ to frame /a, the point Pi 
will be observed in frame /a at p2 = (^3i2/2)^ which 
corresponds to the 3D sceije point = (-X'2,l2i ^2)*- 
According to Eq. (2): 



P2 = 



X2 

Z2 



(7) 



The warping of fi by (u,, u,) to form //* is equivalent 
to applying the cdxneta motion (0,T) to the 3D points 
as though they aie all located on the surface $ (i.e., 
with depths Zs(x,y)). Let P, denote the 3D point 
on the surface s which corresponds to the pixel (x, y) 
with depth ^,(£,1/). Then: 



z, 



Zi 



(8) 



After the image waiping, P, is observed in at =: 
(r^t y^y, which corresponds to a 3D scene point P^ 



459 



PAGE 13/17 ' RCVD AT n6 11:20:18 AM [Eastern DayOght r^^^ 



06/D5/2006 10:20 FAX 6123051228 



MUETING RAASCH GEBHARDT 



@014 



Therefore, according to Eq. (2) aad Eq. (8): 

^1 



and therefore; 



Using Eq. (7) we get; 
and tht^refore: 

p^ = ^A + (i-^)(-r) 



(9) 



Eq (9) shows that the 3D motion between and ft 
is not affected by the camera rotation fl, but only by 
its translation T. Moreover, it shows that is on 
the straight line going through ft arid — Therefore, 
the projection of on the image plane (p^) is on the 
straight hne going through th(5 projection of ft on the 
image plane (i.e. , pj) and the projection of —T on the 
image plane (which is the FOE) This mewi» that 
is found on the radial line emerging from the FOE 
towards pa- In other %vords, the motion between the 
legisleted frames and (i.e., — pa) is directed 
towards, or away From, the FOE, and is therefore in- 
duced by the camera translation T 

Tn Fig 2, the optical flow is displayed before and 
after registration of two frames according to the com- 
puted 2D motion parameters of the image region 
which coiresiponds to the wall at the back of the scene. 
The optical flov/ is given for display purposes only, and 
was noi Ust^d in the registration. Alter registration, 
the rotational component of the optical flow was can- 
celed for the entire scene, and all flow vectors point 
towards the real FOE (Fig. 2,c). Before registration 
(Fig 2-b) the FOE mistakenly appears to be located 
clsowheie (in the middle of the &ame). This id dne 
to the ambiguity caused by the rota4iori iiround the 
Y-axis, whidi visually appears as a translation alo^g 
the X-axis This ambiguily is resolved by the 2D r^- 
islration. 

2.4 Computing Camera Translation 

Once the rotation is canceled by the registration 
of the delected image region, the ambiguity between 





... ,fcr ^-M 



Figure 2: The effect of region registration. The real 

FOE is noarked by -|- 

a) One of the fxamefl. 

b) The optical flow between two adjacent iramcB (be- 
fore re^uatiou), oveilayed on Fig. 2. a. 

c) The optical flow aftec 2D legistzation ol the wall. 
The flow is indnccd by pnre camera translation (after 
the cazneza lotatiop was canceled), and points now to 
the collect FOE. 

d) The computed depth map. Bright regions oorr^ 
Bpond to dose objects 



image motion caused by 3D rotation and that caused 
by 3D translation no longer exists Having only cam- 
era translation, the flow fleld is directed to, or away 
from, the FOE- The computation of the 3D transla- 
tion therefore becomes over determined and numeri- 
cally stable, as the only two unknowns indicate the 
location of the FOE in the image plane. 

To locate the FOE, the optical flow beiween the 
regiatered frnmcis ie computed, and the FOE is located 
using a search method similar to that deecribed in [3 8]. 
Candidates for the FOE are sampled over a half sphere 
and projected onto the image plane. For each such 
candidate, a global error measure is computed from 
local deviations of the flow Held from the radial lines 
emerging from the candidate FOE. The search process 
is repeated by refining the sampling (on the sphere) 
around good FOE candidates. After a few refinement 
iterations, the FOE is taken to be the candidate with 
the smallest error. 

Since the problem of locating the FOE in a purely 
Iranslaiional flow field is a highly overdetermined 
problem, the computed flow fleld need not be accurate. 
This is apposed to mofit methods which try to com- 
pute the ego-motion from the flow Held, and reQuixe 
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ao accurate 9.ow field in order to resolve the rotation- 
tianslation ambiguity. 

2.5 Computing Camera !Rotation 

Let (a,6,c, d,e,/,i?,A) be the 2D motion param- 
etets of the 3D planar surface corresponding to the 
detected image region, as expressed by Eq- (5). Given 
these 2D motion parameters and the 3D translation 
parameters of the camera [Tx^Ty^Tz), the 3D rotar 
tion parameters ol the camera {^Xi • ^z) (as well 
aa the planar surface parameters {a^fi.f)) can be ob- 
tained by solving Eq. (6), which is a set of ^ight linear 
equations in six unknowns. 

iFrom our expeti^^ice, the parameters g and A in 
the pseudo 2D projective transformation, computed 
by the method described in Sec 4, are not as reliable 
as the other six parameters (a, 6, c, ti, /), as and h 
are second order terms in Eq. (5). Therefoie, when- 
ever possible (when the set of Eq. (6) la numerically 
oveidetermincd), we avoid using the last two equations 
(for g and h), and only the first six. This yields 
moze accurate results. 

As a matter of fact, the only case in which ail eight 
equations of (6) must be used to recover the camera 
rotation is the case when the camera translation is 
parallel to the image plane (i-e , T ^ 0 and 2V = 0). 
This is the only coi^figuration of camera motion in 
which the first six equations of (6) do not suffice for 
retrieving the rotation parameters. However, if only 
the first six equations of (6) are used (i.e., using only 
the reliable parameters a, 6» d, e, /, and disregarding 
the unreliable ones, g and />), then only Qz can be re- 
covered in this case. In order to recover the two other 
rotation parameters, Qx and Qy* the second order 
terms (/ and h must be used. This means that for the 
case of an existing translation with Tz ^ D. only the 
translation paraiiiieters (TxjTy/rz) and one rotation 
parameter, Qz (the rotation around the optical axis), 
can be recovered accurately. The other two rotation 
paiamebera, and Qy, can only be approximated. 

in all other configurations of camera J3iotion the 
camera rotation can be reliably recovered 

2.6 Experimental Results 

The camera motion (In cm) between the two 
frame9inFig.2 was: (Tx^Ty^Ti,) = (1.7,0.4, 12) and 
i^x^^Yi^z) = (0**i -1-8*', -3**). The computation of 
the 3D motion parameters of this camera (after setting 
Tz = 12) yielded; (Tx^Ty.Tz) = (1.68,0.16, 12) and 
{Hx.Qy^^z) = (-0-05^-1.7^-3.25°) 

Once the 3D motion parameters of the camera 
a« computed, the 3D scene structure can be recon- 
structed using a scheme similar to that suggested in 



[11). Correspondences between small image patches 
(currently 5x5 pixels) are computed only along the 
radial lines emerging from the FOE (taking the rota- 
tions into account). The depth rrjap is computed from 
the magnitude of these displacements. La Fig. 2.d, the 
computed inverse depth map of the scene (z^~j^) 
displayed. 

3 Camera Stabilization 

Once the ego-motion of the camera is determined, 
this information can be used for post-imaging stabi- 
lisation of the sequence, as if the camera bad been 
mounted on a gyroscopic stabilizer. 

Tot example, to make perfect stabilization, the im- 
ages can be warped back to the original position of the 
first image to cancel the computed 3D rotations. Since 
rotation ia depth-independent, such image warping is 
easy to perform, resulting in a new sequence which 
contains only 3D translations, and looks as if taken 
from a stabilized platform. An example of such sta- 
bilization is shown in Fig- 3.d . Alternatively, the ro- 
tations can be filtered by a low-pass filter so that the 
resulting sequence will appear to have only smooth 
rotations, but no jitter. 

4 Computing 2D Motion of a Planar 
Surface 

We use previously developed methods [15, 16] in 
order to detect an image region corresponding to 'a 
planar surface in the scene with its pseudo 2D- ptojeo- 
tive transformation Tho£c methods treated dynamic 
scenes, in which there were assumed to be multiple 
moving planar objects The image plane was seg- 
mented into the differently moving objects, and their 
2D image motion parameters were completed. 

In this work we use the 2D detection algorithm in 
order to detect a single planar snrlace and its 2D image 
motion parameters Due to camera translation, planes 
at different depths or orientations will have different 
2D motions in the image plane, and will therefore be 
identified as dilTerently moving planar objects. When 
the scene is not piece wise planar, but contains planar 
surfaces, the 2D detection algorithm still detects the 
image motion of its planar regions. 

In this section we describe very briefly how the tech- 
nique for detecting multiple moving planar objects 
locks onto the one planar object and its 2D motion 
parameters. More details appear in [15, 16] 

The projected 2D image motion (t/(4r,f/),v(^,y) of 
a planar moving object in the scene can be approxi- 
mated by the 2D parametric trajisfbrmation of Eq. (5). 
If the support R of this planar object were known in 
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Figure 3i Camera Stabilisation. 

a) One of the Itamefi in the sequence. 

b) The ave^ag^ oi two frame*, having both rotation and 
tranElation. The 'white lines display the image motioii. 

c) The average o{ ihc two framea alter rcgisiratlon oi 
the shiit- Only eRccU oi camera tianslation remain. 

d) The average of the two frames afttt recovering the 
ego-Tvtotion, and Cancelikig the camera rotation. This 
lesijlts in a stabilized pair of imagea. 

the image plane» then it would bo simple to estimate 
its 2D parainctnc image motion (ii,t?) between two 
successive flames, and 7(ii:,5^,t -f 1). This 

could be done by computing the eight parameters 
(a,6, c,</, Cj /, p, A) of the transform ation (see 
Eq (5)) which minimize the following error function 
over the region of support R [jl6]: 

(10) 

The error minimization is performed iteraiively using 
a G&ussian pyramid [4, 15^ 16]. 

Unfortunately, the re^on of support of a pla- 
nar object is not known in advance. Applying the er- 
ror minimization technique to the entire image would 
usually yield a meaningless result. 

Hus, however, is not true for simple 2D irans- 
lations^ where the 2D motion can be expressed by 
(u(a?,y),v(ir,y)) = (a,d). It was shown in [5] that 
the motion parameters of a single translating image 
region can bc recovered accurately by miniinizing the 



error function £?rrO)(G, d) = E(fl;.ir)("-^» + ^^v ^t)^ 
with respect to a and d over the entire image (again, 
using iterations on a multiresolution data structure). 
This can be done even in the presence of other mov- 
ing objects in the region of analysis, and with no prior 
knowledge of their regions of support. This object 
is called the dominant translaiinff ohje^d^ and its 2D 
translation the dojninant £D irajislalion. 

In [15, 16] this method was extended to compute 
kigfifir order 7D motions (2D afJine, 2D projective) of 
a single planar object .among differently moving ob- 
jects. A segmentation step, which marks the region 
corresponding to the computed dominant 2D motion, 
was added- This is the region of the dominant planar 
object in the image. 

The scheme for locking onto a single planar object 
and its 2D image motion is gradual, where the com- 
plexity of dhe 2D x^otion model is increased in each 
computation step, and the ecgxnentation of the pla- 
nar object is refined accordingly More details can 
be found in [16] The 2D motion models used in the 
gradual locking on a planar object are listed below in 
increasing complexity: 

1. Translation: 2 parameters, u(r,y) = a, 
v{x,y) = d This mode) is applied to the cniirc- 
image to get an initial motion estimation. This 
computation is follow5:d by segmentation to ob- 
tain a rough estimate of the object's location 

2. Affine: 6 parameters, u{x,y) = a -i- 6x -f c^j 
«(*! y)~d-hex'\-fy This model is applied oi>ly 
to the segmented region obtained in the transla- 
tion computation step, to get ati affinc approxi- 
mation of the object's motion. The previous seg- 
mentation is leiincd accordingly. 

3. A Moving planar surface (a pseudo 2D pro- 
jective transformation): S parameters [1, 3] (see 
Eq. (5)), u(x,y) = ci + i^a: + cy + ^a;^ + hxy, 
v(ar, 2/) = d 4- car + + gxy hy^. This model is 
applied to the previously segmented region to fur- 
ther refine the 2D motion estimation of the planar 
object, and its segmentation 

5 Concluding Remarks 

A method is for computing ego-motion in static 
scenes was introduced At first, an image region cor- 
responding to a planar surface in the eceoie is de- 
tected, and its 2D motion parameters between suc- 
cessive frames are computed The 2D transformation 
is then used for.image warping, which cancels the rota- 
tional component of the 3D camera motion for the en- 
tire scene, and reduces the problem to pure 3D trans- 
lation The dD translation (the FOE) is computed 
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from the registered frames, and then (Jie 3D rotation 
ia computed by flolving a small set of linear equations. 

It was shown that the ego-motion can be recovered 
reliably in ajl cases, except for two: The case of aij 
entirely planar scene, and the case of an ego-motion 
with a translation in the x-y plane only. The first case 
cannot be uniquely resolved by humans cither, due to 
a visual ambiguity. In the second case it was shown 
that only the translation pajameters of the camera and 
the rotation around its optical axis can be recovered 
accurately. Th^ panning parameters (rotation around 
the X and y axes) can only be roughly estimated in 
this special case. In all other configurations of camera 
motion the ego-motion can be reliably recovered. 

The advantage of the presented technique ia b its 
simplicity, and in the robustness and stability of each 
computational step. The choice of an initial 2D mo- 
tion model enables efficient motion computation and 
numerical stability. There axe no severe restrictions 
on the ego-motion or on the structure of the environ- 
ment Most steps use only image intensities, and the 
optical flow is used only for extracting the FOE in tlie 
case of pure 3D translation, which does not require ac- 
curate optical fiow. The inheient problems of optical 
flow and of feature matching are theorefore avoided 
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